Repository navigation
Verify jupyter_env in a sandbox the agent never touched - #1579
surajsharan wants to merge 2 commits into
Conversation
openenv#1233 took verification off the notebook kernel, and pinned what it does not fix: E2B starts every command as a login shell and the kernel runs as root, so a profile the agent writes still decides what a failing verify command reports. A task can now pass submit_paths to reset. With paths declared, the submission is verified in a second sandbox: a fresh one from the same template, the task's own setup first, then the declared paths copied in as data under /submission, then the verify commands and the reward file, then the sandbox is killed. The order is the boundary. Setup runs before the agent's work exists, so the task's test files cannot be replaced, and /submission is not the working directory and not on PATH or PYTHONPATH. The list defaults to empty and an empty list keeps today's behaviour, so existing tasks score exactly as they did. Paths are relative to /home/user. An absolute path, one that leaves the workspace, a symlink, a socket, a fifo, a device, or a submission past 200 files or 10 MiB is refused with a reason on the state. A file a shell or a test runner reads before the first assertion is left behind even when a declared directory holds it: .bash_profile, .bashrc, .profile, sitecustomize.py, *.pth, conftest.py, pytest.ini, tox.ini and pyproject.toml. Naming one of those exactly still submits it, since that is the task author's decision rather than an accident. The regression that matters is the case openenv#1233 pinned at 1.0: a cell writes an EXIT trap into the profile, and the same episode now scores 0.0 because the verifier never reads that sandbox. The cheating conftest.py @zuraiz-anjum reproduced on coding_tools_env is covered too, and it scores 1.0 again if the exclusion is removed. Verification outside the agent's sandbox is what openenv#1232 asked for, for this environment.
|
Thanks for carrying the case names over, that makes the One gap in
I'll use the same list in |
@zuraiz-anjum measured this on openenv#1579: a failing run reports success through setup.cfg, .pytest.ini, pytest.toml and .pytest.toml the same way it does through pytest.ini, because each can carry addopts = --co, which collects the tests without running them. Confirmed here on pytest 9.1.1: a directory with one failing test exits 0 with any of them. Add those four, and .bash_login, which a login shell reads when there is no .bash_profile. A test pins the whole list so it cannot quietly shrink, and asserts that naming one exactly is still the task author's decision.
|
Good catch, and your table holds on pytest 9.1.1 as well as 9.2. One failing test in a directory,
Pushed all four, plus Thanks for measuring it rather than listing it. |
Summary
#1233 took
jupyter_env's verification off the notebook kernel and pinned what that does not fix: E2B starts every command as a login shell and the kernel runs as root, so a profile the agent writes still decides what a failing verify command reports. A test in that PR asserts the hole rather than hiding it, and #1232 tracks closing it.This closes it for
jupyter_env, as the first environment. On #1232 @burtenshaw asked which task-declared paths may be submitted and which setup and test files must come exclusively from the verifier; this implements the answer I gave there. @zuraiz-anjum then reproduced two of the attacks it refuses, againstcoding_tools_env, and is taking that environment next with the same layout. The design has not been signed off, so treat the shape as proposed rather than agreed, and the RFC note below stands.A task passes
submit_pathstoreset. With paths declared, submitting verifies in a second sandbox:setupcommands, before any of the agent's work exists./submissionas data. It is not the working directory and not onPATHorPYTHONPATH.The order is what makes it a boundary rather than more hardening: setup runs before the submission arrives, so nothing the agent submits can replace the task's test files, and the reward file exists only where the agent has never been.
submit_pathsdefaults to empty, and an empty list keeps the current in-place behaviour untouched, so every existing task scores exactly as it does today. The opt-in is also why the extra sandbox per scored rollout is acceptable.What is refused
Paths are relative to
/home/user. Refused with a reason recorded instate.submission_refused: an absolute path, one that leaves the workspace with.., a symlink, a socket, a fifo, a device, or a submission past 200 files or 10 MiB. Symlinks are reported rather than followed because following one copies whatever it points at.Files that a shell or a test runner reads on the way to the first assertion are left behind even when a declared directory contains them:
.bash_profile,.bashrc,.profile,sitecustomize.py,*.pth,conftest.py,pytest.ini,tox.ini,pyproject.toml. Declaring one of those exactly still submits it, since naming it is the task author's decision rather than an accident.One decision worth flagging
Because the submission lands in
/submission, a task that opts in writes its verify commands against that path. The alternative, copying into the workspace at the same relative paths after setup, would keep existing verify commands working unchanged, at the cost of letting a declared path collide with a file the task's own setup created. I chose the separate tree because that collision is exactly what the boundary is for, and because it matches Harbor's layout, where/testsis the verifier's and the agent's work is not. Say the word if adoption cost matters more and I will switch it.Closes #1232 for
jupyter_env.coding_tools_envis @zuraiz-anjum's next, with the same layout.Type of Change
Alignment Checklist
.claude/docs/PRINCIPLES.mdand this PR aligns with our principles.claude/docs/INVARIANTS.mdand no invariants are violatedbash .claude/hooks/lint.shand tests and addressed all issuesRFC Status
The implementation is environment-local: a
resetkwarg, a second sandbox and one new module underenvs/jupyter_env/server/. No core API, and nothing for another environment to import. The pattern is the part others would copy, so if you would rather have an RFC first, this PR is its reference implementation and I am happy to write it up.Test Plan
New tests in
tests/envs/test_jupyter_environment.py, driving the realJupyterEnvironmentandE2BSandboxagainst two stand-in sandboxes: a kernel whose namespace persists between cells, a process runner that starts each command asbash -l -cwithHOMEset the way envd does, and a file API over each sandbox's own temporary directory.trap 'exit 0' EXITinto the profile, and the episode now scores 0.0, because the verifier never reads that sandbox. Both tests live in the file, so the before and after sit next to each other.conftest.pywhosepytest_runtest_makereporthookwrapper marks every outcome passed scores 0.0 with the refusal recorded, and a rewritten copy of the task's test file does not replace the one setup wrote.conftest.pycase score 1.0 again, so the test fails for the reason it claims.../outside.txtandwork/../../outside.txtare refused while the rest of the submission still scores./etc/passwdis refused and never reaches/submission.pytest tests/envs/test_jupyter_environment.py→ 40 passed. Full suite with CI's dependency set andGITHUB_ACTIONS=true: 4132 passed, 132 skipped, 0 failed.ruff checkandruff format --checkonsrc/andtests/pass,usort checkflags only the two files it flags onmain, andscripts/sync_env_docs.py --checkpasses.Out of scope
find -printf, which is GNU find, as E2B's code-interpreter template has. A template with BusyBox find would need a different listing command, and the stand-in tests cannot catch that.terminus_envandcoding_tools_envstill verify in the agent's kernel. @zuraiz-anjum has thecoding_tools_envport, and Update Terminus env harness and sandbox backends #729 rewritesterminus_env's sandbox.Claude Code Review
N/A